Cost, Statistics, Measures, and Standards for Digital Reference Services: A Preliminary View

THISPAPER REPORTS O N WORK FROM TWO STUDIES I N PROGRESS related to assessing digital library reference services and developing standards that support such services. The paper suggests that two types of standardsutilization and technical-should be considered together in the costing, statistics, and measures for digital reference services. The digital reference community has the opportunity to embed quality standards and assessment data into software and infrastructure by linking utilization and technical standards early in the evolution of digital reference markets. Such an approach would greatly enhance the collection and analysis of a range of cost data related to digital reference service. 1.INTRODUCTION This paper outlines the current status of standards (both utilization and technical) in digital reference with special attention given to issues of cost; both costs incurred by adopting standards, as well as means of assessing cost in digital reference. The article represents preliminary results of a study to develop methods to assess the quality of digital reference services and ongoing work to develop technical standards in digital reference. The Information Institute of Syracuse at Syracuse University and the Information Use Management and Policy Institute at Florida State University conducted the first study. This study is developing digital reference measures; testing and refining these measures and quality standards to R. David Lankes, 621 Skytop Road, Syracuse, NY 13244 Melissa Gross, 1112 Ivanhoe Road, Tallahassee, FL 32312 Charles R. McClure, Francis Eppes Professor and Director, Information Use Management and Policy Institute, School of Information Studies, Louis Shores Building, Rm. 226, Florida State University, 'hllahassee, Florida 32306-21 00 LIBRARYTRENDS, Vol. 51, No. 3, Winter 2003, pp. 401-413 02003 The Board of Trustees, University of Illinois 402 LlBRARY TRENDS/WINTER 2003 describe digital reference services; and producing a guidebook that describes how to collect and report data for these measures and standards. This study began at the October 2000 Virtual Reference Desk (VRD) Conference in Seattle, where the growing digital reference community identified assessment of qiiality as a top research priority. As patrons demand more services online, and as reference librarians seek to better meet patrons’ information needs through the Internet, it has become essential to determine common standards quality. Library administrators need strong, grounded metrics and commonly understood data to support digital reference services, assess the success of these services, determine resource allocation to services, and determine a means for constant improvement of digital reference wiithin their institutions. Project information about this effort can be found at http://quartz.syr.edu/quality/. The second source for this article comes from ongoing work to develop technical standards in digital reference. This work is represented by the development of the Question Interchange Profile (Lankes, 2002) and the newly initiated work of NISO (National Information Standards Organization) Standards Committee AZ (NISO, 2002). This work responds to an increasing call by vendors and technical service staff for clear guidelines to ensure interoperability. Project information about this and related standards efforts can be found at http://wMnu.niso.org/. While, at first, utilization and technical standards may be seen as separate, this paper argues that both, tightly coupled, are essential for the advancement of digital reference and to truly capture a holistic picture of cost. While utilization standards may determine formulae and approaches to determining the total cost of digital reference, technical standards both impact this cost (through tool development or software acquisition), as well as provide a means of distributing and/or recouping these costs. For example, in a consortia, setting a per-question cost can be determined. Properly developed technical standards can “carry” this cost with the question (for example, by providing a field with a dollar figure), greatly easing accounting and enabling the creation of a “question economy” where consortia members can bid on questions or do automated routing to the most cost-effective answer source. These concepts will be expanded below. 2. A DIGITALREFERENCEPRIMER For the purposes of this paper, digital referencp is defined as human-intermediated assistance offered to users through the Internet. Today, libraries are offering a range of human-intermediated reference services over the Internet at an increasing rate. Research byJoe Janes and his colleagues (Janes, 2000) found that 45 percent of academic libraries and 12.8 percent of public libraries offer some type of digital reference service. These services are often ad hoc and experimental. Janes and McClure (1999) found that, for quick factual questions, librarians using only the Web answered a sample of‘ LANKES ET AL./STATISTICS, MEASURES, AND STANDARDS 403 questions as well as did those using only print sources. Many libraries conduct digital reference service in addition to existing obligations with little sense of the scale of such work or its strategic importance to the library. This paper does not provide a comprehensive review or analysis of digital reference and digital reference services. Gross, McClure, and Lankes (2002) have published elsewhere a detailed analysis of digital reference literature. Despite this and other such reviews, there is limited knowledge about costs, assessment, and standards related to digital reference seivices. As the studies discussed in this paper are completed, one product will be a manual to assist librarians assessing digital reference services on a range of criteria and measures (McClure, et al., 2002). 3. DEVELOPING OF STANDARDS A TYPOLOGY IN DIGITALREFERENCE The authors divide digital reference standards into two types: 1. Utilization: Those standards that deal with the use and delivery of digital reference services, specifically to determine whether a digital reference service is succeeding. These can include a mix of qualitative and quantitative metrics as well as more abstract statements on best practices or objectives for a service. 2. Technical: The use of hard tools (software, hardware, protocols, and other standards enforced by computers with little or no interpretive room) and soft tools (primarily metadata and organizational schema where aspects of human description are controlled, but still open to interpretation). These two high-level categories have been further refined in two separate efforts. It should be noted, however, that both of these efforts are ongoing, and these refinements may change. 3.1. REFINING UTILIZATION STANDARDS The first effort to refine the digital reference typology is the “Assessing Quality in Digital Reference Services” conducted by the Information Institute of Syracuse at Syracuse University and the Information Use Management and Policy Institute at Florida State University (Lankes, et al., 2001). This study is supported by OCLC and the Digital Library Federation and a wide range of library organizations (see Table 1) This study has compiled a preliminary set of metrics, statistics, and standards for assessing digital reference from a review of the literature and a series of site visits (http://quartz.syr.edu/quality/VRDSiteVisitsummary.pdf). These measures were reviewed by the study’s advisory committee (made up of the primary sponsors and the sustaining members), and revised. As of this writing the revised measures are being field tested in a variety of library types (federal, academic, and public). 404 LIBRARY TRENDS/WINTER ZOO3 Table 1. Members of the Quality Studv. Sustaining Membera (hntributiiw Membei-s Multnoniah (:ounty Library (the first McKeldin Library, University of Maryland public library to join the study) Mid York Library System The Library of Congress Bristol University, University Library Strozer Library, Florida State Univer-sity Liverpool John Moores University Cleveland Public Library University Library, Syracuse University Pcnnsylvania Office of ConiInonwealth Library of Michigan Libraries, Bureau of Library Development State Library of Florida, Division of Library and Information Services Reference and User Senices Association 3.1.1. QUALITY STANDARDS Utilization standards can be first refined into performance measures and quality standards. A quality standard is a specific statement of the desired or expected level of performance that should be provided regarding a senice or some aspect of that service. A quality standard can be measured to determine the degree to which that standard is in fact being met (Kasowitz, et al., 2000).A quality standard defines the level of performance that an organization is willing to accept for a particular service or activity. Quality standards are important because they: Encourage library staff and administration to discuss and agree upon what constitutes “quality” for a specific senice; Provide clear guidance as to the expected quality that a particular service or activity should offer; Educate staff-and especially new staff-as to the expected quality of service that should be provided; Recognize that there may be differing acceptable levels of quality for different aspects of digital reference services; and Provide a basis for rewards and demonstrating/reporting accountability. Quality standards are not performance measures. A performance measure might be “correct answer fill rate” whereas the quality standard might be “the digital reference service will have a correct answer fill rate of 65 percent.” The assessment study specifically states that there is no “correct” standard for any specific digital reference service. The correct standard will rather depend on the goals and objectives of the library, the amount of resources that can be committed to reaching a particular standard, local situations affecting digital reference services, and the relative importance of one quality standard versus another. For one library, an awareness level of digital reference services of 30 percent among faculty (for example) may be acceptable; for another, the standard might be 60 percent. LANKES ET AL./STATISTICS, MEASURES, A N D STANDARDS 405 While not specifically spelling out all possible quality standards, the study proposes six Quality Standards that appear to span specific circumstances and domains: 1. Courtesy: The behavior of the library or institution’s staff. 2. Accuracy: The “correctness” of answers provided by a digital reference staff. 3. Satisfaction: Users’ determination of their success in interacting With the digital reference service. 4. Repeat Users: The percentage of users that reuse a service after first encounters. 5. Awareness: The population user group’s knowledge that the service exists. 6. Cost: The cost per digital reference. It is assumed that each of these standards will have a strong qualitative component. However, to fully define these standards, the study created five types of performance measures that can be used to better determine success in meeting quality standards: 1. Descriptive Statistics and Measures: Statistics and measures to determine the scale arid scope of a digital reference service. 2. Log Analysis: Statistics that can be derived from analysis of logs generated by Web and digital reference software packages. 3. User Satisfaction Measures: Statistics and metrics seeking to understand the user view of a digital reference service. 4. Cost: Measures that gage outlay of financial resources to run an ongoing digital reference effort. 5. Staff Time Expended: Measures to determine staff time dedicated to digital reference. Each of these classes of measures is then further refined into specific metrics and statistics as seen in Table 2. Further refinement within these measures is also possible. For example, the assessment study has associated data collection methods to each measure, but such refinement is too specific for the discussion in this paper. Nonetheless, special attention should be given to the cost measures and standards. 3.1.2.COSTMEASURES AND STANDARDS The economics of reference is an area that has long been neglected. Indeed, the economics of information in general has only recently received significant attention (Kingma, 2001).Assigning costs to reference service is a complicated task but one that must be faced in order to realistically assess the true costs of doing business, to make assessments about the most efficient ways to provide services, and to determine how to share the costs of this service in setting up and participating in collaborative service models. 7izbL 2. Utilization Standards bv Class.